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Background 

Functional  brain  imaging  has  progressed  far  beyond  just  identifying  and  cataloguing  the  brain  areas 
that  activate  in  a  task.  As  of  about  6  years  ago,  this  project  proposed  to  bring  a  new  technology,  fMRI,  to 
the  service  of  understanding  spatial  cognition.  The  task  has  involved  developing  new  methodologies, 
mastering  a  new,  evolving  technology,  and  synthesizing  the  several  sub-disciplines  of  neuroimaging, 
neuroanatomy,  and  neurophysiology.  But  as  the  report  below  indicates,  the  progress  has  been 
considerable.  We  are  obtaining  important  results,  publishing  the  studies  in  first-rate  journals,  and 
presenting  the  data  in  talks  to  diverse  audiences.  The  results  from  this  project  and  many  others  are  being 
incorporated  within  the  developing  theory,  so  that  the  progress  is  more  than  a  collation  of  findings,  but  an 
integrated  body  of  knowledge  with  underlying  principles  for  extensibility  and  application. 
Correspondingly,  the  theory,  methodological  approach,  and  the  ongoing  projects  are  beginning  to 
influence  the  fields  of  cognitive  psychology  and  cognitive  neuroscience. 

1.  IMAGERY,  SPATIAL  WORKING  MEMORY,  AND  COGNITIVE  STRATEGIES 

Volumetric  modulation  and  dynamic  recruitment  of  neural  activation  One  of  the  cortical 
network  properties  that  we  have  extensively  explored  is  the  volumetric  modulation  of  the  brain  activation 
as  a  function  of  the  cognitive  processing.  For  example,  in  studies  requiring  the  mental  rotation  of  an 
object,  the  amount  of  fMRJ-measured  activation  increases  with  angular  disparity  (Carpenter  et  al.,  1999  in 
J.  of  Cognitive  Neuroscience).  We  interpret  the  volumetric  properties  as  reflecting  the  ratio  of  the  task’s 
demand  for  cognitive  resources  to  the  supply,  or  capacity  utilization.  Similarly,  in  studies  requiring  the 
mental  tracking  of  the  location(s)  of  one  or  three  targets  in  a  2-D  or  3-D  visual  array,  the  volume  of  the 
brain  activation  increases  both  with  the  number  of  items  to  be  tracked  and  the  dimensionality  of  the  space 
(3-D  elicits  a  greater  volume).  Moreover,  the  two  effects  (that  of  number  of  targets  and  dimensionality) 
are  not  localized  to  single  parts  of  the  brain,  but  occur  in  both  the  prefrontal  cortical  regions  and  the 
parietal  regions,  indicating  extensive  collaboration  between  the  members  of  the  spatial  network.  Several 
articles  on  volumetric  modulation  during  spatial  thinking  have  been  published  or  are  in  press. 

Imagery.  One  very  recent  fMRI  study  examined  how  people  mentally  rotate  a  3-dimensional  object 
(an  alarm  clock)  that  is  retrieved  from  memory  and  rotated  according  to  a  sequence  of  auditory 
instructions.  We  manipulated  the  geometric  properties  of  the  rotation,  such  as  having  successive  rotation 
steps  around  a  single  axis  versus  alternating  between  two  axes.  The  latter  condition  produced  much  more 
activation  in  several  areas.  Also,  the  activation  in  several  areas  increased  with  the  number  of  rotation 
steps.  During  successive  rotations  around  a  single  axis,  the  activation  was  similar  for  rotations  in  the 
picture  plane  and  rotations  in  depth.  The  parietal  activation  (plausibly  one  of  the  sites  of  geometric 
transformation  computations)  was  very  similar  in  this  situation  (in  terms  of  location  and  volume)  to 
mental  rotation  of  a  visually  presented  object  (in  a  Shepard-Metzler  task).  However,  there  was  much  less 
activation  in  the  extra-striate  (ventral-stream)  areas  in  this  clock  task  (where  there  is  no  visual  input)  than 
in  a  Shepard-Metzler  task  (where  much  more  analysis  has  to  be  done  on  a  perceptually  available  figure). 
The  findings  indicates  that  mental  rotation  may  be  decomposable  to  some  extent  into  the  figual  analysis 
portion  and  the  geometric  transformation  portion,  and  that  the  parietal  and  extrastriate  areas  respectively 
underpin  these  two  types  of  processes.  (At  the  same  time,  functional  connectivity  analyses  (in  a  submitted 
manuscript)  indicate  that  there  is  a  higher  degree  of  synchronization  between  these  two  areas  during  the 
recognition  of  rotated  familiar  objects).  So  the  specialization  of  function  co-exists  with  the  interaction  of 
function,  a  property  of  cortical  processing  that  is  incorporated  into  4CAPS  models.  The  article  describing 
these  results  are  to  be  found  in  Just,  Carpenter,  Maguire,  Diwadkar,  &  McMains,  2001 . 

Strategy  differences.  Most  complex  problems  and  even  some  seemingly  simple  ones  admit  the  use 
of  more  than  one  strategy  or  processing  mode,  such  as  verbal  vs.  visuo-spatial.  With  the  recent  advent  of 
neuroimaging  techniques,  it  is  now  possible  to  examine  the  cortical  systems  that  support  these  modes  of 
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processing,  and  to  understand  how  their  behavioral  characteristics  relate  to  their  neural  substrates.  In  fact, 
it  is  possible  to  tell  which  mode  a  person  is  using.  We  taught  participants  to  do  a  sentence-picture 
verification  task  using  either  a  verbal  strategy  or  a  visual-spatial  strategy.  One  objective  was  to  examine 
the  relation  between  the  strategies  (i.e.,  cognitive  routines)  and  their  underlying  patterns  of  cortical 
activation  using  functional  Magnetic  Resonance  Imaging  (fMRI),  building  on  existing  data  which 
suggested  partial  separation  between  the  cortical  systems  responsible  for  linguistic  processing  and  those 
responsible  for  visual-spatial  processing.  A  second  objective  was  to  examine  how  the  strategy-related 
differences  in  cortical  activity  are  modulated  by  individual  differences  in  cognitive  skill. 

The  verbal  strategy  produced  more  activation  in  language-related  cortical  regions  (e.g.,  Broca’s 
area),  whereas  the  visual-spatial  strategy  produced  more  activation  in  regions  that  have  been  implicated  in 
visual-spatial  reasoning  (e.g.,  parietal  cortex).  One  way  of  identifying  which  strategy  or  mode  is  being 
used  is  to  count  the  number  of  activated  voxels  in  two  regions,  Broca’s  area  and  the  left  parietal  cortex, 
and  then  subtract  the  latter  from  the  former.  This  difference  should  be  larger  for  the  verbal  than  visual- 
imagery  strategy  because  the  verbal  strategy  tends  to  activate  Broca’s  area,  whereas  the  visual-imagery 
strategy  tends  to  activate  the  left  parietal  cortex.  This  procedure  eorrectly  identifies  the  strategies  used  for 
ten  out  of  twelve  (83%)  participants. 

These  relations  were  also  modulated  by  individual  differences  in  cognitive  skill:  individuals  with 
better  visual-spatial  skills  (as  measured  by  the  Vandenberg,  1971,  a  mental  rotation  test)  had  less 
activation  in  the  left  parietal  cortex  when  they  used  the  visual-spatial  strategy.  These  results  indicate  that 
language  and  visual-spatial  processing  are  supported  by  partially  separable  networks  of  cortical  regions, 
and  suggests  one  basis  for  strategy  selection:  the  minimization  of  cognitive  workload.  This  work  by 
Reichle,  Carpenter,  and  Just  appeared  in  Cognitive  Psychology. 


2.  METHODS  DEVELOPMENT 

Functional  connectivity.  To  progressively  make  the  mapping  closer  between  cognition  and  brain 
activation,  it  is  necessary  for  us  to  continually  develop  new  techniques  and  software  for  analyzing  the 
fMRI  data.  For  example,  a  key  network  property  that  we  have  been  examining  is  cdWcA  functional 
connectivity  between  activated  brain  areas;  it  is  assessed  by  the  correlation  of  the  activation  time-series 
data  between  voxels  in  different  areas,  a  time  series  with  an  observation  every  few  seconds.  The  general 
assumption  is  that  the  functioning  of  voxels  whose  activation  levels  rise  and  fall  together  is  coordinated. 
In  one  analysis,  the  correlations  are  based  on  only  those  periods  when  the  task  is  being  performed 
(excluding  the  fixation  periods),  so  that  the  time  series  indicates  the  momentary  fluctuations  in  activation 
level  during  the  actual  task  performance.  This  is  a  novel  measure  and  contrasts  with  examining  the 
correlation  throughout  the  entire  time  course  of  a  study,  a  measure  which  is  primarily  dominated  by  the 
alternation  of  the  task  with  rest  periods.  Restricting  the  analysis  to  only  the  task  periods  essentially 
examines  the  micro-structure  of  the  processing  fluctuations.  Our  results  show  that  during  a  spatial 
tracking  task,  the  functional  connectivity  (i.e.  degree  of  synchrony)  between  the  activated  voxels  in 
DLPFC  and  the  parietal  regions  reliably  inereases  with  the  processing  load.  We  have  several  papers 
published  or  in  press  that  report  the  results  of  functional  connectivity  and  more  in  the  pipeline.  For 
example,  we  have  found  that  the  functional  coimectivity  between  the  parietal  area  and  DLPFC  increases 
with  cognitive  workload,  indicating  closer  coordination  among  areas  when  the  cognitive  demands  are 
higher  (Diwadkar  et  al.,  2000). 

Assessing  the  functional  coimectivity  among  the  nodes  of  a  large-scale  neural  network  moves  fMRI- 
imaging  toward  an  analysis  of  cognition  in  terms  of  dynamic  systems.  To  perform  these  analyses,  we 
(Vlad  Cherkassky)  have  developed  a  package  called  VOXCOR  which  flexibly  performs  many  types  of 
correlational  analyses  with  appropriate  visualization  tools  to  display  time  courses  of  activation  in  various 
cortical  areas. 

Increasing  temporal  resolution  in  fMRI.  Even  though  fMRI  measurement  involves  a  delay 
between  the  oceurrence  of  a  given  thought  process  and  the  hemodynamic  response,  it  is  nevertheless 
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possible  to  methodologically  and  technologically  strengthen  the  relation  between  the  measurement  and 
the  theoretical  account.  We  are  continuing  to  explore  higher  temporal  and  spatial  resolution  on  a  new 
fMRI  scanner,  which  has  higher  capabilities  than  were  previously  available.  We  started  this  work  by 
pushing  the  limits  of  an  older  scanner,  and  discovering  linguistic  effects  on  brain  activation  within  about 
1200  msec  of  stimulus  onset,  long  before  the  hemodynamic  response  reaches  its  maximum  value  at  4-6 
sec  (Carpenter  et  al,  1999  in  Neuroimage).  With  the  new  scanner,  we  are  able  to  sample  30,000  voxels 
(each  one  about  30  mm^)  once  every  500  msec,  and  still  obtain  excellent  sensitivity.  We  are  currently 
exploring  sampling  each  voxel  every  250  msec.  The  higher  sampling  rate  does  not  overcome  the 
hemodynamic  delay  (which  has  to  be  at  least  500  msec),  but  it  does  provide  a  more  frequent  and 
measurement  of  the  activation.  This  increased  spatio-temporal  resolution  brings  the  fMRI  data  into  closer 
correspondence  with  the  theoretical  account  of  the  ongoing  cognitive,  as  specified  within  the 
computational  modeling  system  being  developed.  Within  4CAPS  models,  the  quantum  is  about  50  msec, 
but  the  next  level  of  aggregation,  at  about  200  or  250  msec,  is  perhaps  even  more  cognitively  meaningful. 
Thus  the  theory  and  fMRI  data  are  approaching  the  similar  levels  of  temporal  granularity. 

3.  DEVELOPMENT  OF  A  THEORY  AND  A  COGNITIVE  MODELING  SYSTEM 


As  the  relation  between  brain  activation  and  cortical  organization  begins  to  unfold  in  the  fMRI 
research,  it  is  being  incorporated  into  the  design  of  4CAPS,  which  is  a  production-system  architecture 
with  several  connectionist  features  first  reported  in  the  lust  et  al.,  1999  article  in  Human  Brain  Mapping. 
The  theory  has  recently  been  described  in  a  comprehensive  manuscript  that  has  been  prepared  for 
publication.  It  is  clear  that  the  theory  and  4CAPS  provide  a  powerful  conceptual  tool  for  understanding 
current  results  and  guiding  future  research.  The  existence  of  the  computational  modeling  system  makes  it 
possible  to  ask  precise,  concrete  questions  about  performance  in  tasks  that  have  not  yet  been  examined 
with  fMRI,  some  of  which  can  never  be  examined  with  fMRI  because  of  the  technology’s  restrictions  on 
motion  and  magnetic  and  electrical  activity. 

The  computational  system  makes  it  possible  to 
develop  models  of  a  particular  cognitive  task.  One 
example  is  a  model  of  sentence  comprehension  that 
accounts  for  the  word-by-word  processing  times,  the 
probability  of  comprehension  error,  and  the 
modulation  of  the  brain  activation  in  several  cortical 
areas  over  the  course  of  about  24  sec.  The 
accompanying  diagram  depicts  each  model 
component  as  a  block,  with  each  block  corresponding 
to  a  cortical  region.  The  6  blocks  correspond  to  the  6 
left-hemisphere  components  of  the  comprehension 
model.  The  3  lexical  components  are  placed  within  a 
larger  box.  The  CONSTRUCT  and  INTERPRET 
components  are  divided  horizontally  into  four  main 
parts,  corresponding  to  the  four  kinds  of 
representations  the  model  deals  with:  lexical, 
syntactic,  semantic,  and  goal.  The  arrows  are 
intended  to  show  the  interaction  between  Input  to  the  system  consists  of  the  perceptual 

components.  It  is  difficult  in  a  static  diagram  to  encoding  (orthographic  or  phonological)  of  one 

depict  the  adaptive,  dynamic  behavior  of  each  word  of  a  sentence  at  a  time 

component,  as  well  as  the  dynamic  allocation  of 

work  between  the  components.  Some  of  the  predictions  of  this  model  are  presented  in  Part  4  (on 
development  of  deconvolution  methods).  Another  example  of  a  new  4CAPS  model  is  one  that  solves  the 
Tower  of  London  problems.  The  models  provide  the  tools  to  quantify  and  predict  cognition  and  brain 
activation. 
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Besides  the  new  theory-building,  there  still  remains  considerable  interest  in  the  previous  version  of 
the  cognitive  architecture  (3CAPS),  namely  the  form  that  preceded  the  brain  activation  modeling, 
accounts  of  working  memory  constraints  and  individual  differences.  In  a  commentary  soon  to  appear  in 
Psychological  Review,  Just  and  Varma  describe  how  the  symbolic  and  connectionist  mechanisms  within 
the  hybrid  3CAPS  architecture  combine  to  produce  a  processing  style  that  provides  a  good  match  to 
human  sentence  comprehension  and  other  types  of  high-level  cognition.  (This  article  should  not  be 
contused  with  the  manuscript  describing  the  4CAPS  theory).  The  article  relates  the  properties  of  3CAPS 
to  other  connectionist,  symbolic,  and  hybrid  systems. 

On  another  theoretical  front,  we  have  written  an  overview  article  in  Current  Opinion  in 
Neurobiology  that  characterizes  working  memory  and  executive  fimction  (Carpenter,  Just,  &  Reichle, 
2000).  The  paper  reviews  the  neuro-imaging  literature  and  argues  against  a  localization  view  of  working 
memory  and  executive  function.  We  argue  that  the  existing  meta-analyses  of  empirical  data  on  verbal 
and  spatial  working  memory  and  specific  empirical  studies  addressing  the  localization  hypothesis  have 
failed  to  support  localization.  In  its  place,  we  suggest  the  usefulness  of  the  perspective  of  dynamic 
systems  theory  to  understand  aspects  of  the  generativity  and  organization  of  human  cognition. 
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objects  retrieved  from  memory:  An  fMRI  study  of  spatial  processing.  Journal  of  Experimental 
Psychology:  General. 
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and  brain  function.  Human  Brain  Mapping.  8, 128-136. 

Reichle,  E.  D.,  Carpenter,  P.  A.,  &  Just,  M.  A.  (2000).  The  neural  basis  of  strategy  and  skill  in  sentence- 
picture  verification.  Cognitive  Psychology.40,  261-295. 

Just,  M.  A.,  Carpenter,  P.  A.,  Maguire,  M.,  Diwadkar,  V.,  McMains,  S.  (2001).  Mental  rotation  of 
objects  retrieved  from  memory:  An  fMRI  study  of  spatial  processing.  Journal  of  Experimental 
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Symposium  at  Psychonomics  99  in  Los  Angeles.  One  of  us  (MJ)  organized  a  symposium  on  the 
contributions  that  functional  brain  imaging  is  making  to  the  understanding  of  high  level  cognition.  The 
formal  title  was  "Functional  Brain  Imaging  and  Higher  Level  Cognition."  The  presenters  were  Jordan 
Grafman,  Ed  Smith,  Morton  Gemsbacher,  Dan  Schacter,  and  Marcel  Just. 
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Invited  Presentations  by  Co-PI’s  during  the  tenure  of  the  current  ONR  grant 

Cognitive  loading  and  the  future  military.  Invited  presentation  at  the  Cognitive  Sciences  Workshop 
sponsored  by  the  Science  Applications  International  Corporation  (SAIC),  January  8-9,  1998,  Strategic 
Assessment  Center,  McLean,  Virginia. 

Working  memory  and  working  brain:  Computational  and  fMRI  studies  of  cognition.  Invited 
presentation  at  the  University  of  Pittsburgh,  Western  Psychiatric  Institute  and  Clinic,  February  19,  1998, 
Pittsburgh  Pennsylvania. 

Modeling  neural  function  and  high-level  cognition.  Invited  presentation  at  CNBC/ICN  Workshop, 
Mellon  Institute,  Carnegie  Mellon  University,  March  26, 1998,  Pittsburgh,  Pennsylvania. 

fMRI  and  higher  cognitive  functions.  Invited  presentation  at  the  First  Annual  Meeting  of  the 
NICHD/NIDCD  Collaborative  Programs  of  Excellence  in  Autism  (CPEA),  June  14-15, 1998, 

Washington  DC. 

Recovery  of  language-related  brain  function  after  aphasia.  Invited  presentation  at  the  James  S. 
McDonnell  Foundation  Program  in  Cognitive  Rehabilitation,  September  10-12, 1998,  St.  Louis,  Missouri. 

fMRI  and  the  Neuro-architecture  of  cognition.  Invited  presentation  at  the  National  Institute  of 
Mental  Health  “Cognitive  Neuroimaging  Research:  Design  and  Interpretation  Meeting”,  September  IS¬ 
IS,  1998,  Rockville,  Maryland. 

Cognitive  load  distribution  in  normal  and  autistic  individuals.  Invited  presentation  at  the  29* 
Carnegie  Symposium  on  Cognition,  “Mechanisms  of  Cognitive  Development:  Behavioral  and  Neural 
Perspectives”,  October  1 1,  1998,  Pittsburgh,  Pennsylvania. 

Modeling  cognitive  systems  with  artificial  intelligence  &  fMRI.  Invited  presentation  at  BrainMap 
’98:  Human  Brain  Mapping  and  Modeling,  December  7,  1998,  San  Antonio,  Texas. 

Language  in  the  brain:  fMRI  studies  of  sentence  comprehension.  Colloquium  presented  at  the 
University  of  Pittsburgh,  Department  of  Linguistics,  April  7, 1999,  Pittsburgh,  Pennsylvania. 

fMRI  and  the  neuro-architecture  of  cognition:  studies  of  sentence  comprehension.  Colloquium 
presented  at  the  Center  for  Cognitive  Sciences,  University  of  Minnesota,  May  13,  1999,  Minneapolis, 
Minnesota. 

Language  in  the  brain:  fMRI  studies  of  sentence  comprehension.  Invited  presentation  at  the 
International  Conference  on  Basic  Mechanisms  of  Language  and  Language  Disorders,  University  of 
Leipzig,  September  30, 1999,  Leipzig,  Germany. 

Dynamic  cortical  systems  subserving  cognition:  What  fMRI  reveals  about  (some)  secrets  of  the 
mind.  Colloquium,  Department  of  Psychology,  November  8,  1999,  Carnegie  Mellon  University, 
Pittsburgh,  Pennsylvania. 
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the  Collaborative  Program  of  Excellence  in  Autism  (CPEA)  Meeting,  September  25-29,  2000,  Denver, 
Colorado. 

fMRI  studies  of  the  architecture  of  language  comprehension.  Keynote  address  presented  at  the  64*'’ 
Conference  of  the  Japanese  Psychological  Association,  November  6-8, 2000,  Kyoto  Japan. 
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